Multisensory wound simulation

ABSTRACT

A Tactile-Visual Wound (TVW) simulation unit is a physical device designed to help simulate a wound on a human being or human surrogate (e.g., a medical manikin) for instructing a trainee to learn or practice wound-related treatment skills. For the trainee, the TVW would feel (to the touch) like a real wound, look like a real wound when viewed using an Augmented Reality (AR) system, and appear to behave like a real wound when manipulated.

BACKGROUND OF THE INVENTION 1. Field of the Invention

This invention relates to medical simulations. More specifically, itrelates to a system for combining tactile and visual wound (TVW)simulations for medical training and education.

2. Brief Description of the Related Art

Combat life savers, combat medics, and medical corpsman are the firstresponders of the battlefield, and their training and skill maintenanceis of preeminent importance to the military. While the instructors thattrain these groups are highly competent, the simulations of battlefieldwounds are constructed from simple static moulage including fake bloodthat instructors apply immediately prior to training. The simplicity ofthe presentation often requires the instructor to describe the wound orremind the trainee during an exercise about the qualities of the woundthat are not represented in the wound presentation, including how thewound is responding to treatment. This added effort decreases time thatcould be spent providing instruction, increases noise into the cognitiveprocess, and interferes with the flow of the trainee experience. Eventhese simple simulations take time and effort to create, set up, andmanage, before and during the training exercise.

The preparation time and overall compressed schedule of a trainingcourse means that trainees get limited hands-on practice in realisticsettings. While dynamic computer-based simulations have been availablefor some time, virtual training in front of a 2D computer screen or evenin a full virtual reality environment does not provide the trainee withthe same experience as hands-on training with moulage wounds on physicalbodies. Typical field training is even more simplistic: the state of theart is still a “casualty card” that tells a “casualty” actor how toportray a wound specified on the card, and the trainee sees at beststatic moulage and fake blood as a portrayal of the wound.

Existing simulated wound systems fall into two main categories. First,there exist purely physical wound models such as those made from coloredrubber. These are static, both visually and behaviorally. Second, thereexist virtual wound models rendered by computer graphics. These can lookrealistic when viewed through a tracked stereo head-worn display forexample, but because they are purely virtual they cannot be manipulatedas physical entities.

There are several challenges with current headset/HMD-based augmentedreality systems on the market, portable or wearable systems, includingtheir form-factor (many are bulky and uncomfortable), their limitedcomputing power, and (perhaps the most apparent) their typically limitedfields of view. All these limitations make for a constrained immersiveexperience that must be accounted for when designing an augmentedreality application, especially one meant for immersive training.

Virtual wound systems could simulate some degree of tactile sensationvia actuated tactile gloves. What is needed in the art is moving thetactile aspects back into the wound (as with the physical wound models),thus supporting manipulation via un-adorned hands, while stillsupporting the dynamic virtual aspects.

BRIEF SUMMARY OF THE INVENTION

Augmented Reality (AR), especially the proliferation of wearable ARheadsets, glasses, or head-mounted displays (HMDs), has the potential torevolutionize casualty care training in both military and civiliancircumstances. AR provides a unique mix of immersive simulation with thereal environment. In a field exercise, a trainee could approach acasualty role-player or mannequin and see a simulated wound rendered inan AR headset appear registered on the casualty. This combination ofhands-on, tactile experience with the real world on the one hand, andsimulated, dynamic wounds and casualty responses on the other, has thepotential to drastically increase the realism and overall quality ofmedical training. A system that can additionally provide automatedinstruction could enable anytime/anywhere training and reduce the burdenon instructors.

The present invention is composed of several technologies focused onenhancing the multi-sensory training experience. This includes ARtechnology to enhance the visual aspects of training—portraying woundsin ways that not only look more accurate but also exhibit the dynamicsof real wounds, including their progression over time and theirresponses to treatment. This includes leveraging the moulage that isused today to provide the passive haptic sensations of wounds, whileextending this moulage with active features. These enhancements includeusing embedded actuators that can provide more realistic haptic effectsassociated with pulse, breathing, and specific wound features such asdischarging liquid that—when combined with visuals—is perceived asblood; and embedded sensors to facilitate dynamic responses to thetreatments the trainee is administering to the wound.

A TVW is shaped to match the size and shape of a wound, e.g., apuncture, stab, slice, tear, abrasion (i.e., road-rash), lacerations, oravulsions. The device may be pliable or may include pliable componentsthat support the device being affixed to a curved real or simulatedhuman (or animal) body surface, such as an arm or leg. The outer surfaceof the device is made to feel like skin, for example using asilicon-based material. In an embodiment of the invention, the outersurface of the device is covered with markers or patterns that aredetectable (observable) to a sensing system, but generally imperceptibleto a human, to facilitate dynamic computer vision (camera) or othersensor-based localization in 3D position and 3D orientation, as well asshape deformation estimation, e.g., to estimate the device shapedeformation resulting from the adherence to a curved body part, ordynamic manipulation by a trainee (e.g., squeezing a wound to close anopening). The markers or patterns could be passive, e.g., painted orcolored material, or active, e.g., using LEDs, photosensors, magneticcoils, or other active electronic components that facilitate thelocalization and deformation estimation. In an alternative embodiment ofthe invention, two layers of markers overlay where the material itselfpartially opaque allowing for some additional estimation of deformationof the wound. This may be achieved by either differences in deformationof the two layers or changes in the interference pattern between to thetwo layers.

The device may contain sensors, e.g., pressure sensors distributed overor within the device, e.g., to measure forces applied by a humanpracticing the application of pressure to stop bleeding, e.g., to affectvisual and physiological simulations, or inertial or magnetic sensors tohelp in estimating the device position and orientation. For example, anembodiment of the invention may detect pressure by resistive,capacitive, or other material affixed inside or on the bottom of asimulated physical wound. The device may contain actuators or sources,e.g., to emit sounds, smells, or liquids; or to produce or simulatesmall movements associated with the wound, e.g., tremors or agitation.The device may contain or be associated with a processor that could,e.g., perform estimation, control, computation, and communication. Thedevice may contain or include wireless communication capabilities suchas Bluetooth or similar. The device may be powered by batteries, e.g.,rechargeable batteries with an inductive charging capability, orconnected directly to an external power source.

When applied to a simulated patient (real human or manikin) andactivated, the AR system continuously tracks the TVW with respect to theAR headset (glasses or HMD), estimates the deformation, measures thesensors (applied pressure, etc.), renders (via computer graphics) anappropriate dynamic view of the simulated wound, and affects allappropriate actuator outputs. For example, the trainee might see (inhis/her AR headset) what appears to be blood flowing out of the wound,as the vital signs “drop,” then as increasing wound pressure is appliedby the trainee (e.g., as determined by the pressure sensors), theapparent rate of the blood loss (as rendered by the AR system) wouldslow, and the physiological simulation would reflect stabilized vitals,etc. Real-time depth or other models of the trainee's hands, medicaldevices, etc. could also be used to affect the simulated visualsgenerated by the AR rendering system. For example, blood could be madeto appear to flow out of the wound and over the hands, until appropriatepressure is applied at which time the blood flow would halt, and theremaining blood would stabilize, etc. A liquid reservoir and pump may beaffixed on or within the corporeal object, the pump communicativelycoupled to the computer processor, wherein liquid from the reservoir ispumped to and emitted by the wound, to simulate bodily liquid consistentwith the simulation. The liquid may be chroma-keyed (e.g., blue orgreen) so that the computer processor may recolor and/or texturize thefluid in alignment with the simulation. Alternatively, or inconjunction, the pump or secondary pump may atomize fluid to createsmells consistent with the simulation.

An embodiment of the invention could model various wound characteristicssuch as eschar tissue or different levels of depth in a burn wound,e.g., a blast casualty injury compounded with burns. In yet anotherembodiment, a 3D model of the corporeal object is accessible by thecomputer processor wherein the software process reconstructs the spatialand dimensional parameters of both the visual graphic enhancements andthe 3D model of the corporeal object for the augmented reality mechanismwherein the optical image of the corporeal object is spatially anddimensionally aligned to the 3D model wherein the 3D model may berendered opaque or semi-transparent via the augmented reality mechanism.One or more glove indicia on one or more trainee-worn gloves engage thecorporeal object and simulated wound, the computer processor furtheraccesses a 3D glove model wherein the glove indicia convey theorientation, location and articulation of trainee hands to dynamicallyalign the 3D glove model with the trainee hands. The computer processormay chroma-key the trainee-worn glove so that their optical imageopacity may be modified.

BRIEF DESCRIPTION OF THE DRAWINGS

For a deeper and more complete understanding of the invention, referenceshould be made to the following detailed description, taken inconnection with the accompanying drawings, in which:

FIG. 1 is an enhanced photographic image of a military medic simulationaccording to an embodiment of the invention.

FIG. 2 is a partially sectional, side-elevation, partially diagrammaticview of an embodiment of the invention.

FIG. 3 is a diagrammatic view of an embodiment of the invention.

FIGS. 4a-c are side elevation views of a simulated casualty wherein limbarticulation changes not only the AR wound positioning but state aswell.

FIGS. 5a-b are front elevation views of a simulated casualty whereinlimb articulation changes not only the AR wound positioning but state aswell.

FIG. 6 (left) shows an augmented reality character of a virtual childwith a nose wound in a concept HOLOLENS application; (center) a userwearing a HOLOLENS inside the Human-Surrogate Interaction Space (HuSIS)at the University of Central Florida Synthetic Reality Lab (SREAL); and(right) an ARDUINO board equipped with an HC-06 Bluetooth shield.

FIG. 7 shows an illustration of an image pattern approach for a circulargunshot wound, i.e., patterning the wound and periphery with several(possibly distinct) images.

FIG. 8 is an illustration of the border tracking approach according toan embodiment of the invention.

FIG. 9 is an illustration of image markers in the corners of the wound.

FIG. 10 are illustrations of background development focused on positiontracking with infrared and depth information, which could be adapted forhybrid position-orientation tracking with an AR HMD.

FIG. 11 shows ARToolKit markers and inventors' prototype implementationon the HOLOLENS.

FIG. 12 shows illustrations of two rendering/transformation pathwaysthat deal with either tracking loss in the HOLOLENS' camera or in theHOLOLENS′-based self-tracking.

FIG. 13 an example of multi-marker tracking using an embodiment of theinvention.

FIG. 14 shows results of the comparison between VUFORIA (blue) andARToolKit (green). The error bars show the standard deviation. Thex-axis shows the actual distances; the y-axis shows the trackeddistances.

FIG. 15 shows a plot of the relative differences (offset errors) betweenthe tracked distances and their actual distances.

FIG. 16 shows overall frames-per-second for VUFORIA (blue) and ARToolKit(green). Both of them did not reach 30 fps. VUFORIA did not provide anytracking data for 1.66 m and beyond. ARToolKit provided more data overlonger distances.

FIG. 17 is an illustration of the rotation directions showing theARToolKit marker. Inventors started from an orientation in which themarker was facing straight at the HOLOLENS in full view. Inventors thenrotated the marker around one of the three axes in space. The marker wasrotated by two full 360-degree rotations around each of the three axes(i.e., X, Y, Z).

FIGS. 18 and 19 compare results for the ARToolKit (“ARTK”) versus theIMU for pitch. Each plot shows the distance (in meters) to the HOLOLENSon the x-axis, and the angular error (in degrees) of the tracking datapoints on the y-axis.

FIGS. 20 and 21 compare results for the ARToolKit (“ARTK”) versus theIMU for roll.

FIGS. 22 and 23 compare results for the ARToolKit (“ARTK”) versus theIMU for yaw.

FIGS. 24 and 25 compare angular errors between the ARToolKit (“ARTK”)versus the IMU for signed pitch in the range from −180 to +180 degreesfrom the actual orientation (measured by OPTITRACK).

FIGS. 26 and 27 compare angular errors between the ARToolKit (“ARTK”)versus the IMU for roll in the range from −180 to +180 degrees from theactual orientation (measured by OPTITRACK).

FIGS. 28 and 29 compare angular errors between the ARToolKit (“ARTK”)versus the IMU for yaw in the range from −180 to +180 degrees from theactual orientation (measured by OPTITRACK).

FIG. 30 is a plot showing results of marker size evaluation for a 50 mmmarker. The x-axes show the actual distances of the markers, while they-axes show the tracked distances.

FIG. 31 is a plot showing results of marker size evaluation for a 40 mmmarker. The x-axes show the actual distances of the markers, while they-axes show the tracked distances.

FIG. 32 is a plot showing results of marker size evaluation for a 30 mmmarker. The x-axes show the actual distances of the markers, while they-axes show the tracked distances.

FIG. 33 is a plot showing results of marker size evaluation for a 20 mmmarker. The x-axes show the actual distances of the markers, while they-axes show the tracked distances.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 shows a photographic image of a military trainee 10 and simulatedcasualty 45 wherein AR glasses (or AR headset/HMD) 60 and sensors 66 ontrainee 10 render wounds 65 over static moulage on casualty 45. Aportable computer 50 on the trainee's body processes data from thecasualty 45 and trainee 10 to enhance the simulation. For example,vitals may be rendered in a heads-up display through the AR glasses 60on the trainee 10 or on a wrist-wearable display 70. As trainee 10performs medical procedures on casualty 45, casualty 45 sends data toportable computer 50. The data may be responsive to several sensorsincluding pressure, limb manipulation, temperature, light, and the like.The sensor data is processed on portable computer 50 wherein the woundsand associated visuals 65 rendered in AR glasses 60 change in accordancewith the sensor data.

For example, casualty 45 detects pressure over simulated wound 65.Portable computer 50 then modifies AR rendering 65 to decrease or stemthe simulation of blood flowing out of the wound. Embodiments of thepresent invention may further add more depth to this simulation. A fluidreservoir and pump inside the casualty 45 may respond to instructionsfrom portable computer 50 to increase, decrease or stop the flow offluid used to simulate blood pressure within the vascular system andchanges in pressure throughout a training simulation. One embodiment of“simulated blood” includes the rendering of virtual blood visually,while optionally emitting a very slight amount of liquid (e.g., water orGlycerol) to give a tactile sensation of the visually apparent “blood.”The liquid does not necessarily have to match real blood in terms ofconsistency, color, or amount to potentially still be effective.Therefore, trainee 10 does not just have the visual rendering of thewound, but also the tactile and temperature kinesthetic feedback fromthe bodily fluid.

Continuing along the same example, should trainee 10 stem the blood flowthen the vitals rendered in AR glasses 60 or display 70 are modified byportable computer 50 consistent with decreasing blood loss. Yet anotherenhancement may include audio feedback. This may be from casualty 45itself having an embedded speaker or the audio may be sent toheadphones, headsets, earbuds or separate speaker for trainee 10. Theaudio may include groans, speech or the like stemming from casualty 45.The audio may also be used to affect tactile or haptic sensations thatare felt not heard.

In the event the pressure applied to the wound is excessive (as detectedby the pressure sensor in or on casualty 45) then audio feedbackindicating pain is sent according to logic followed by portable computer50. By the same token, pain or other conditions may induce shaking ormovement within a real patient. Therefore, casualty 45 may include avibrational motor controlled by portable computer 50. Such a motor mayinclude an eccentric rotating mass vibrational motor, a linear resonantactuator, or the like.

An advantage of the invention is the multi-sensory approach to bothvirtual reality enhancements and to non-computer-enhanced aspects suchas fluids, movement, and temperature. Going back to the example of painsimulation, not only could casualty 45 tremble from the vibrationalmotors, the AR glasses could render a face upon casualty 45 grimacing inpain. Responsive to the actions of trainee 10 to mitigate the pain, thecasualty's computer-generated face could soften its features to showcasualty 45 is more comfortable.

In FIG. 2, a trainee 10 manipulates an artificial limb 40. Trainee 10wears an AR headset 60. Headset 60 is communicatively coupled tocomputer 50. Computer 50 may be portable, non-portable, on premise orremotely located but communicatively coupled to local components.Trainee manipulates limb 40 with trainee hands 20 and 30. Indicia 130,131, 132, and 133 on limb serve as reference points for headset 60 torender AR wound enhancements to trainee 10. In this example, the forearmof limb 40 is wounded and trainee 10 seeks to stem the flow of simulatedblood 120. To enhance the simulation, fluid reservoir/pump 80 iscommunicatively coupled to computer 50 wherein computer 50 may causepump to push simulated blood or other fluid to wound in forearm. Trainee10 senses the fluid not only visually but by its tactile and temperaturechange through hand 20. Pressure sensor 90 is communicatively coupled tocomputer 50. Responsive to increased pressure by hand 20 on wound,computer 50 may adjust the amount of fluid pushed through pump 80.Sensor 100 may detect limb articulation and/or elevation of forearm.Thus, sensor 100 communicates to computer 50 that trainee 10 haselevated forearm and computer 50 then adjusts the amount of fluid pushedthrough pump 80.

Vibrational motor 110 is communicatively coupled to computer 50 whichdirects motor 110 to actuate responsive to the simulation of tremors ordiscomfort.

Vital sign display 70 is communicatively coupled to computer 50 whichassimilates data received from sensors, trainee 10, and the simulationscenario itself to present vital signs indicative of the patient'sstatus. The vital sign display 70 may be rendered to headset 60 or maybe an external display visible to the trainee 10 and/or supervisinginstructors.

In FIG. 3, trainee 10, casualty 45, and simulation engine 200 arecommunicatively coupled through data bus 210. Simulation engine 200processes functions and procedures responsive to sensor feedback 240from casualty. Array of sensors 260 may detect proximity, pressure,temperature, sound, movement, and light. Data bus 210 may becommunicatively coupled by wired cables or wireless protocols.Simulation engine 200 accesses simulation database 220 which stores oneor more simulations. Logic in simulation engine 200 varies thesimulation through several means including transmitting modifications tothe augmented reality output 280 to trainee 10, modifying vitals 10 ofcasualty 45 and generating, modifying, or ceasing physical reactions 250on or within casualty 45. Array of simulation reactions 270 includemovement, sound, smell, vibration, fluid flow, and temperature.

In an embodiment of the invention, trainee vitals 290 may be received bysimulation engine 200 which modifies the simulation based on the stateof the trainee 10. For example, trainee vitals 290 indicate trainee 10has an elevated pulse rate from the simulation. This may causesimulation engine 200 to modify the simulation to create a dirersituation for casualty 45 to further test the ability of trainee 10 todeal with stress. In yet another embodiment of the invention providessupervisor user interface 230 that allows a trainer to override certainautomated aspects of the simulation engine to test and directly observethe competency of trainee 10 and/or further instruct trainee 10. Forexample, supervisor user interface 230 could “freeze” the simulation sothat an instructor can step into the simulation and assist the trainee10 in a procedure or technique.

In FIG. 4a casualty 45 rests on his back. Indicia are affixed on hisleft cheek 400, upper left arm 401, lower right arm 402, right thigh 403and left lower leg 404. For the purposes of this illustration, indicia400-404 are visually discernable. However, indicia may be invisible tothe human-detectable light spectrum and detected in other wavelengthssuch as UV or IR reactive dyes such as described in U.S. Pat. No.8,653,445, or magnetic or otherwise imperceptible. This may beadvantageous when casualty 45 is a manikin that is sought to berealistic without the distractions of visual indicia. As notedpreviously, the indicia establish the topology of the casualty 45 sothat AR graphic enhancements may be computationally sized, orientated,positioned, and warped as the trainee and/or casualty 45 is moved,engaged, responds or is otherwise deformed. Simulated AR wound 300 islocated on the upper left arm and simulated AR wound 310 is on the lowerleft arm. As the left arm is raised in FIG. 4b , the AR rendering ofwounds 300 and 310 are repositioned, orientated, and warped insynchronization with the physical embodiment of casualty 45. Anembodiment of the invention applies physiological or other relevantlogic to casualty 45's wounds responsive to treatment. In the case ofFIG. 4c , the left arm is slightly lowered with respect to FIG. 4b andAR rendering of wound 310 is reduced in intensity to wound 311 on thelower left arm. The AR rendering of wound 300 on the upper arm in FIG.4b is intensified as shown in FIG. 4c as wound 301. Thus, the trainee'sarticulation of the left arm produces immediate effects on the ARsimulation which dynamically adjusts to physiologic parameters. Alsoshown in FIG. 4c are indicium 405 on the upper right leg and indicium406 on the lower right leg with wound 320 rendered on the right knee.

In an alternative embodiment of the invention, mechanical sensors detectthe articulation of the limbs and communicate data to the AR generationapplication to size, orient, place, and warp graphic enhancements at theappropriate location. In yet another alternative embodiment of theinvention, shape detection of the anthropomorphic shape of the casualtyitself obviates the need for separate indicia.

In FIG. 5a-b , indicia 410-411 are affixed to the chest of casualty 45.Indicium 420 is affixed to upper right arm. Indicium 421 is affixed tolower right arm. In FIG. 5a , right leg orientation is straight and ARrendered knee wound 321 is displayed. However, in FIG. 5b , right leg isbent which would physiologically open a wound on the knee by pullingback skin from the thigh and shin. Accordingly, AR rendered wound 320 islarger in size in FIG. 5b than knee wound 321 displayed in Figure Sa.Thus, an embodiment of the present invention does more than simplyreorient the spatial and dimensional features of the AR rendering overcasualty 45 but also modifies the AR rendering in accordance with thephysiology of treatment of the casualty 45. In addition to changes inthe AR rendering, tactical, auditory and other sensory feedback may begenerated. Additional examples include, but are not limited to,increased bleeding, decreased bleeding, changes in vital signs, auditoryoutput, olfactory output (e.g., the smell of burnt flesh), respiratorymovement changes, limb movements, and the like.

The present invention may be integrated with medical simulationphysiological engines such as those provided under the BIOGEARS andPULSE PHYSIOLOGY ENGINE brands to provide bidirectional communicationbetween the casualty 45, the trainee, and extrinsic peripherals such asvital signs monitors.

It should be noted that in addition to a manikin, this technology may beapplied over a living human subject, e.g., a role player or standardizedpatient. For example, a silicon moulage with sensors and tactilefeedback components (e.g., pumps, vibrational components, etc.) may beaffixed to the limbs, head, torso or extremities of a living subject forthe simulation. This may reduce the size and cost of the simulationequipment and allow for greater portability. Obviously, sensors andresponse actuators would not be embedded in the human subject but withinthe moulage temporarily affixed to the subject during the simulationexercise. Reference indicia may be removable adhesive stickers on theliving subject or simply affixed to the removable moulage.

Incision and Puncture Embodiment

In an embodiment of the invention, a manikin includes a surfacecomprised of skin-like material that may be cut with a scalpel orinjected with a syringe giving full, tactile feedback to the trainee.The manikin surface may be comprised of a self-healing polymer materialinfused with nickel atoms which not only reconnect after separation, butinterpret altered electrical flow from touch, cut and injections tolocalize the area of engagement. Any number of self-healing materialsmay be used. For example, a fluorocarbon-based polymer with afluorine-rich ionic liquid may be used wherein the polymer networkinteracts with the ionic liquid via highly reversible ion-dipoleinteractions, which allows it to self-heal. Additional variationsinclude polypyrrole (PPy) nano wrapped in the cellulose surface andimmobilized Fe3O4 magnetic particles; rubber particle reinforced epoxyorganic nano self-repair; and nickel dendrites and bristle graphene.Pressure sensing may be done by a resistive or capacitive material putinside or on the bottom of the wound.

Embodiments of the invention may be tactile or non-tactile. For example,an embodiment employs a “virtual scalpel” or “virtual syringe” thatcreates indicia detectable by a camera (on the headset or otherwise)when used in a manner imitating the real device. The indicia are visibleor invisible to the trainee but would invoke the AR simulation localizedin that area. In a rudimentary form, the “scalpel” could be a green dryerase marker. As the trainee drags the marker across some skin thecamera detects the green color and automatically knows that it is anincision. The simulation then creates an AR rendering (e.g., separationof the skin) which the trainee may further engage with. The marker inthis embodiment provides a distinct chroma key for which simulations maybe presented. However, this embodiment is tactile in the sense that thetrainee's marker puts pressure on the surface of the manikin butnon-tactile as to the cutting of the skin. Accordingly, based on theneeds of the simulation, resources available, and training requirements,the level of tactile feedback may be varied according to the presentinvention.

Indicia on the syringe convey the distal tip of the needle lumen so theAR simulation understands its spatial location and orientation. Apressure tip on the syringe fires an injection event and since thesoftware knows where the syringe is it can render AR at the correctinjection site. This embodiment uses optically detected indicia but mayalso convey tactile feedback to the trainee as well.

Thus, simulating cutting or injection in general may be accomplished bytracking surgical devices and/or physical props used as stand-ins forthose devices. Furthermore, an alternative embodiment may be achieved bytracking the props (and therefore knowing the time-varying locations ofthe simulated scalpel edge or syringe needle). Simulated cutting orinjections are therefore accomplished by optical marking/tracing means.In yet another embodiment, simulated cutting or injections may also beachieved by other prop tracking mechanisms (e.g., optical, magnetic,acoustic, inertial, or other means).

An important aspect of using a real scalpel or syringe is the tactilesense of the actual injection of a needle into a skin/body-likematerial, and cutting into something, where the simulated visualscorrespond. Accordingly, an alternative embodiment to self-healingmaterial may include replaceable sheets of silicon “skin” used forcutting, injecting, and suturing according to the invention.

Reduction to Practice

Prototyping efforts focused on a simulation of instrumented moulageincluding visuals presented through an Augmented Reality (AR)Head-Mounted Display (HMD), and a connected device platform allowingbi-directional communication between a miniature computer and a computercontrolling the visuals on the HMD.

A proof-of-concept bi-directional Bluetooth control/effect prototype wasbased on an HMD from Microsoft Corporation under the brand MICROSOFTHOLOLENS and a connected device using an ARDUINO minicomputer equippedwith an HC-06 Bluetooth shield to act as a basic Internet of Things(IoT) device. ARDUINO is a brand-identifier for an open-source hardwareand software company, project and user community that designs andmanufactures single-board microcontrollers and microcontroller kits forbuilding digital devices.

UNITY is a cross-platform game engine developed by UNITY Technologies.The UNITY engine supports more than 25 platforms and is used to createthree-dimensional, two-dimensional, virtual reality, and augmentedreality games, as well as simulations and other experiences. The UNITYgraphics engine was used as the integrated development environment andas the rendering system for the HOLOLENS. To be able to harness theHOLOLENS' Bluetooth capabilities, the .NET 4.6 framework developed andmaintained by Microsoft Corporation was used within the UNITYenvironment. Using .NET 4.6 provided advanced synchronization protocols,which was important to using the HOLOLENS' Bluetooth API in combinationwith the ARDUINO. Using Bluetooth, the device was synchronized with theHOLOLENS, whereas TCP/IP and UDP were used to communicate data to andfrom the connected device. Initial tests included comparing packet lossand latency in different situations and environments, which resulted insubsequent initial optimization efforts of the packeting systems.

We then attached a basic capacitive touch sensor to the ARDUINO. Thetouch data received from that sensor was streamed from the device to theHOLOLENS once the touch such as from a finger (e.g., pressing on awound) was detected. This transmitted data then triggered a state changein the UNITY graphics engine on the HOLOLENS. In this prototype, thevirtual content displayed on the HOLOLENS consisted of a rigged andanimated three-dimensional character of a child (see FIG. 6), which wasavailable through in-house developments at SREAL. The characterappearance and animation were modified to include a nose wound, whichemitted AR blood in varying strengths until the user wearing theHOLOLENS would reach toward the nose and press their finger on the woundto stop the bleeding. In this artificial scenario, the ARDUINO and touchsensor were placed at the approximate location where the nose of the ARchild was located, such that a light touch on the AR wound would stopthe bleeding, whereas lifting the finger from the touch sensor wouldresult in the bleeding to resume (see FIG. 6). FIG. 6 (left) shows an ARcharacter of a virtual child with a nose wound in a proof-of-conceptHOLOLENS application. FIG. 6 (center) shows a user wearing a HOLOLENSinside the Human-Surrogate Interaction Space (HuSIS) at SREAL. FIG. 6(right) shows a for-development ARDUINO board equipped with an HC-06Bluetooth shield.

Alternative embodiments to improve this approach include replacing thetouch sensor with pressure sensors such as from ADAFRUIT or moreflexible pressure-sensitive material such as VELOSTAT, and differentsilicone-based coating materials, into which a miniaturized version ofthe connected device can be embedded. Additional components can beembedded into the moulage to exert physical changes of the shape of theartificial wound and to change the tactile feedback provided whiletouching the moulage such as via small actuators or vibration motorsconnected to the ARDUINO.

Wound Tracking

Using computer vision libraries such as VUFORIA, OPENCV, or VISIONLIB,natural features or pre-placed markers can be tracked by means ofdigital image processing and computer vision algorithms. Various issueswith tracking image-based markers exist, such as the proximity of themarker to the camera, deformations, occlusions, and lighting conditions.Multiple different approaches can address some of these challenges.

Patterning the Wound

This approach involves patterning the wound with several (possiblydistinct) images, which allow an AR HMD with inside-out cameras (such asthe HOLOLENS or META2) to be able to find the wound in a variety ofdeformations and angles. In particular, we make use of a multi-scaleapproach, which means we first try to find a marker in the camera imageconsisting of the whole image #1-9 (see FIG. 7). Conversely, in case ofocclusions, close proximity, or in case the marker is distorted andloses its overall coherence, we can track individual self-sufficientparts. For instance, if the wound is deformed (e.g., wrapped around anarm) and the HOLOLENS can only see #1, 2, and 3, it may still be able tofind these smaller regions.

Border Markers

Instead of patterning the whole wound, one can use multiple largemarkers, e.g., four image markers in the corners of the wound, which theHOLOLENS then can use to track the wound (see FIG. 9). The mainchallenge with this method is that the markers would have to berelatively decent in size to work; and thus, may be impractical fordeveloping a smaller (e.g., gunshot) wound. This approach may also beless desirable than the pattern or border approach (FIG. 8) since theretheoretically could be angles in which no markers are fully visible;meaning the HOLOLENS can lose complete tracking of the wound. It appearsthat the main approach would be to use multiple markers, e.g., 2 markersbordering the wound to the sides, 3 markers arranged in a triangle, 4markers arranged in a rectangular or 4 markers arranged in a circularfashion around the wound in the center. All markers need to be bigenough such that they can be detected by the HOLOLENS. However, there isa trade-off in increasing the number of markers around the woundconsidering the computational cost of tracking multiple markers, e.g.,with the limited computational power of the HOLOLENS, and the spacerequirements.

Model Based Approaches

Traditionally, trying to retrieve the full 6 Degrees of Freedom (DOF)pose of a 3D model at a high framerate is quite challenging sinceobjects can be ambiguous in their pose and can undergo occlusions aswell as appearance changes, and the corresponding computer visionapproaches tend to be computationally expensive. 3D object tracking fromcolor image streams can be roughly divided into sparse methods that tryto track local correspondences between frames, and region-based methodsthat exploit more holistic information about the object such as shape,contour or color. Both directions have pros and cons, e.g., the formerperforms better for textured objects whereas the latter performs betterfor texture-less objects. With the advent of commodity RGB-D sensorssuch as those in the HOLOLENS or KINECT, these methods have been furtherextended to depth images, which is beneficial since image-based contourinformation and depth maps are complementary cues, one being focused onobject borders, the other on object-internal regions. While any solutionusing such model-based tracking in the scope of this project is heavilydependent on the chosen implementation of a computationally lightweightcomputer vision algorithm or use of an optimized (and usually limited)library, several options exist in this direction.

For instance, an embodiment of the invention includes creating arigid-body bracelet which would go around the arm of a simulatedcasualty. The bracelet provides features in the depth and color/infraredstreams of the HOLOLENS such that the bracelet and arm could be trackedfrom an arbitrary angle, i.e., it could be rotated or moved freely. Theidea is that the wound would cover a certain region of the arm and thebracelet is covering a part of the periphery. The use of either onelarge or multiple (geometrically different) smaller bracelets areoptions. In the maximal case, these could be wrapped around the upperarm, lower arm, wrist, etc. Some model-based tracking approaches areimplemented in VUFORIA, VISIONLIB, and OPENCV, which are sufficient totrack such a rigid body. The bracelet could also be patterned so that itcould employ both model-based tracking and image-based tracking. Thisapproach does not seem desirable for wound tracking on predominantlyflat areas of the body, e.g., for a wound on the chest, where a 2Dimage-based marker tracking approach would be optimal. In such areas,adding a 3D model-based marker might improve tracking performance whenthe camera/HOLOLENS is looking at the wound from a steep angle, but themarker would be very noticeable. Using a model-based approach to trackthe wound itself does not seem that useful, since one of the maindesired features is that the wound could deform, which would mean theloss of tracking.

Testing demonstrated that the VISIONLIB API supports tracking a modelthat can take on different “states” of a semi-rigid object (e.g.,bracelet or wound), when it deforms into certain shapes. We investigatedmodel-based APIs and algorithms published at the leading conferences(ISMAR, CVPR, etc.) with regards to RGB-D model tracking.

Hybrid Approaches

Apart from full 6 DOF image and model-based tracking approaches, using ahybrid 3 DOF position and 3 DOF orientation tracking approach has value.Instrumented moulage may include a low-cost Inertial Measurement Unit(IMU) to measure its orientation in space, which would optimally bebased on 9 DOF sensors with 3-axis accelerometers, gyroscopes, andmagnetometers to compensate for drift. If the orientation does not haveto be tracked using image or model-based computer vision approaches,this makes tracking the position using computer vision and RGB or RGB-Ddata easier.

One method includes an infrared LED mounted on a tracked object. The LEDis visible in the infrared camera of a MICROSOFT KINECT, which (aftercalibration) is then used to cast a ray into the scene along which theLED must be located (see FIG. 10). The depth information from the KINECTcould then be used to compute the distance of the LED from the KINECT.Combining the information from the 2D position of the LED in theinfrared camera and the depth of the LED in the depth image then allowedits position to be tracked at the accuracy of the KINECT sensor, which(although not optimal) proved higher than skeleton-based trackingapproaches.

Whenever the AR HMD sees the wound, optimally, it would update itsposition. In parallel, the IMU embedded in the wound could always updateits orientation. To compensate for an inherent drift in IMU data, e.g.,when close to metal objects interfering with the magnetometers, onecould use the orientation data from occasionally tracked image markersand re-calibrate the IMU's orientation. One benefit of this approach isthat it could be used in parallel to image and model-based tracking,e.g., as a fallback solution, with very low computational cost.

FIG. 10 is an illustration of work focused on position tracking withinfrared and depth information, which is adaptable for hybridposition-orientation tracking with an AR HMD.

Prototype Development

Marker Tracking

ARToolKit Implementation

Based on initial tests, we looked into the ARToolKit API for markertracking on the HOLOLENS. ARToolKit is an open-source computer trackinglibrary for creation of strong augmented reality applications thatoverlay virtual imagery on the real world. It is currently maintained asan open-source project.

The HOLOLENS version of ARToolKit proved to be more challenging to workwith than the common version for desktop computers and webcams: trackingframe rates would constantly stagger to under 1 frame per second, itfailed to superimpose a hologram on top of a marker accurately out ofthe box, and some extensive testing was required to deduce the bestconfiguration of ARToolKit.

It appeared that the HOLOLENS was not giving enough computational powerto the tracking routine. Configuring ARToolKit to run without any debugsymbols and using a lower resolution configuration file for ARToolKit'sARUWPControlller improved frame rates yet was nowhere near the author's25 to 30 frames per second performance. After extensive debugging andcode tracing, we traced back the issue to the ARUWPVideo class withinthe ARToolKit scripts. The ARUWPVideo class offers the ability todisplay a “preview” screen of the HOLOLENS' webcam footage in a user'sHOLOLENS environment. This preview is supposed to reduce frame rates,however, it appeared that the exact opposite was happening: within apreview window, frame rates would stagger to what we were experiencing.Casting the preview window media to an empty GameObject within UNITYwould bring performance up to the expected benchmarks.

This might be occurring due to the architecture of the implementation ofARToolKit for the HOLOLENS. When a video preview window is beingdisplayed, webcam footage is being polled so that the preview windowappears to be real time. This does not seem to occur without a previewwindow, and a lack of constant polling on the webcam API seems tostagger the marker tracking routine. The webcam pipeline and the markertracking routine are intertwined considerably: if the webcam isconstantly being polled for updated footage, it forces the markertracking routine to be on-par with the real time footage.

We thus cased the webcam preview to an empty GameObject (a container inthe UNITY Engine), which provided a fix to this issue. The second issuehad to do with the fact that the geometry of the webcam footage does nothave an exact correspondence to the geometry of a HOLOLENS environment.This geometrical correspondence needed to be adjusted by creating acustom web cam calibration file for ARToolKit and by adjustingARToolKit's undistortion function.

We created a custom calibration file using Long Qian'sOPENCV-to-ARToolKit calibration program. Using several photos of a blackand white checkerboard taken at varying angles using the HOLOLENS, thesefiles were fed into an OPENCV calibration file creator, and then fedinto Long Qian's convertor. We created a calibration file at the1344×756 resolution, which provides a wider tracking field of view thanthe default 896×504 resolution at a ˜10 frames per second performancecost. Using this calibration file in conjunction with tuning values inthe undistortion functions produced a hologram that appears quiteaccurately on top of the marker.

The third part was adjusting the ARToolKit settings and configuration,e.g., we ended up converting the webcam footage into mono and fed thatinto ARToolKit instead of using plain colored web cam footage.

FIG. 11 shows an example multimarker setup based on ARToolKit markersand inventors' prototype implementation on the HOLOLENS. With ARToolKit,inventors developed code that positions a hologram between multiplemarkers, referred to as “Multi-Marker Holograms,” and likewise theirmarkers as “Multi-Markers.” Depending on what markers are visible, thecode finds what set of markers can be used to determine a position forthe multi-marker hologram and, if such markers exist, use theirpositions to place the multi-marker hologram in the HOLOLENSenvironment.

In a basic four-marker case, inventors considered a flat square surfacewith markers in the corners (see FIG. 11). Seeing multiple markersimproves the performance of tracking the correct pose and positioningthe hologram relative to the marker(s). In particular, if diagonalmarkers are visible, the orientation accuracy appears to be considerablyhigher. The IMU would be used to adjust the hologram in cases oforientation changes.

Visible Markers and ARToolKit

We made improvements to the ARToolKit implementation for the HOLOLENS,the most notable of which being improvements to the visual calibration.We fine-tuned the undistortion function so that holograms appear alongthe line-of-sight of a physical marker. We also made modifications tothe ARToolKit's marker detection parameters to increase the probabilityof a marker being identified, and to decrease the probability of acamera region being falsely identified as a marker. For instance, welowered the corresponding confidence threshold to 0.4 and used matrixmarker types.

We also worked on our multi-marker detection system. Here, the hologramis placed relative to the position of two or more markers. Thecorresponding code track the markers in parallel and places a hologramusing the position of the most reliable marker as indicated byARToolKit, while keeping a list of all visible makers so that if onemarker goes invisible, the system can place the hologram relative to theother markers. Some aspects of this system are incorporated into oursingle marker system. For instance, to help prevent detection of falsemarkers, all markers know how far apart they should be from each other.This helps markers from being falsely detected when an area in theHOLOLENS' surroundings is falsely identified as being a marker. Ofcourse, this failsafe only works in the case that other markers arecurrently visible.

New Version of ARToolKit

A new version of ARToolKit was released by its developer, Long Qian ofJohn Hopkins University. This new version of ARToolKit was supposed toprovide a steady 30 fps tracking performance at 1344×756 pixelsresolution, which is a considerable improvement over the last version ofARToolKit, which provided ˜25 fps at 844×504. This new version ofARToolKit also supports the IL2CPP backend, which means that oursoftware can support the latest versions of UNITY. We worked on theintegration of this newest version of ARToolKit into our software.

Marker Loss Compensation

In VUFORIA Engine Release v7.5.20, a larger update was made, whichimproved model and image tracking for smaller targets. We tested thisrecent release to see if it would work for our purposes, as VUFORIA is amore mature library than ARToolKit. However, ARToolKit still seemed tobe superior in its tracking ability for more simple markers. Hence, wedecided to not pursue replacing ARToolKit with the latest release ofVUFORIA.

Inventors made several smaller improvements. For instance, we discussedthe integration of the MIXEDREALITY TOOLKIT. Due to compiling issuesthis was temporarily removed. However, we could reintegrate it. Weimproved upon our “debug dialogs” to include more details fordevelopers. We also implemented a Bluetooth “heartbeat,” so that theHOLOLENS software can possess confidence that the smart wound hardwareis on and in reach of sending/receiving data at all times.

FIG. 12 shows illustrations of two rendering/transformation pathwaysthat deal with either tracking loss in the HOLOLENS' camera or in theHOLOLENS' SLAM-based self-tracking (Simultaneous Localization andMapping). M_(MA) is the (static) transform from marker relative to theatificial wound. M_(CD) is (dynamically) determined by ARToolKit whenthe marker is visible in the camera image. M_(DC) (static) is theextrinsic calibration performed through Microsoft's HoloLens applicationprogramming interface (API). M_(WD) (dynamic) is the transform fromWorld Anchor in Microsoft's Spatial Map. The rendering andtransformation pathways include: (1) relative to world anchor(M_(WA)=M_(MA)M_(CD)M_(DC)M_(WD)), persistent even when marker trackingis lost; and (2) relative to display (M_(DA)=M_(MA)M_(CD)M_(DC)),persistent eve when world tracking is lost.

Multi-Marker Hologram Placement

An issue in the development was how we can properly superimpose ahologram in the correct position over a moulage when the placement ofmarkers can vary. To that end, we developed a calibration, whichrequires a user to get all the markers in the view of the HOLOLENS, andthe user then places the hologram on the moulage. The markers thenrecognize where the hologram is placed relative to themselves and usethat information to superimpose the hologram in the correct position asillustrated in FIG. 13. FIG. 13 shows large ARToolKit marker(s) placedon torso to be visible in HoloLens' camera images from a distance;multiple smaller ARToolKit markers placed close to instrumented moulage;and 9-axis Inertial Measurement Unit (IMU) embedded in moulage. Thisrelied on a larger marker for initial tracking, however, it uses smallermarkers when possible and advantageous.

This is easy to implement, as at its core it simply has each markerstore the location of the hologram relative to itself, which can beextended in the future. Our system is based on older features of ourmulti-marker system, such as using a distance interval to prevent falsepositives in the camera-based marker detection from being considered.

Visual Tracking Improvements

We tested and improved the visual tracking approaches. To compare theperformance, we ran marker tracking tests with ARToolKit and, incomparison, ARToolKit performed better than VUFORIA in all of our testruns. ARToolKit was able to track markers as small as 1″ at meaningfuldistances (within a one-meter spherical radius), and while ARToolKitdoes show visual tracking drift when a user's head moves quickly,VUFORIA also showed the same effect. While neither library is perfect,ARToolKit appears to be the library of choice for our application.

We investigated how well ARToolKit performs with marker tracking at“far” distances, meaning more than one meter away from the marker, andif there were any adjustments which could be made to the ARToolKitconfiguration to improve tracking at these distances. One of the initialapproaches we followed to improve tracking at such distances was to runARToolKit's image analysis algorithms on a higher resolution webcamvideo stream. The HOLOLENS provides three webcam resolutions: 896×504,1280×720, and 1344×756. The first resolution is the resolution we havebeen using since our implementation of ARToolKit as it performs thefastest. The 1280×720 resolution, while providing better image quality,offers little in comparison to the 1344×756 resolution, as the latterhas a wider field of view (67 degrees versus 45 degrees), and thealgorithms seemed to run on the latter without a notable performancechange from the former.

Our original hypothesis was that increasing the resolution from 896×504to 1344×756 would improve marker tracking at a distance, at the loss ofupdate rate. In our tests, the different resolutions yielded nonoticeable change in the distance at which ARToolKit could trackmarkers, while resulting in a clearly reduced update rate. The onlynotable advantage of the 1344×756 resolution over the 896×504 resolutionis an increased field of view. Typical performance with the 896×504resolution yielded around 25-30 fps, while the 1344×756 resolutionyielded between 21-27 fps.

ARToolKit Optimization

Inventors optimized the ARToolKit based marker tracking implementationfor the HOLOLENS. As previously discussed, from the tested markertracking libraries, ARToolKit is the one library that is able to trackmarkers that are farthest away from the camera due to the very simplelayout of these markers compared to VUFORIA and other libraries.However, it had multiple issues, mainly related to the low frame ratethat could be reached on the HOLOLENS with the native implementation ofthe ARToolKit for the HOLOLENS that is not optimized for the HOLOLENSand does not take into account the peculiarities of this highlyoptimized hardware and software platform.

Earlier, inventors realized a challenge related to how the HOLOLENS'video streaming implementation for the ARToolKit (specifically theARUWPVideo class) affects performance. Namely, we observed that whenthere is no video preview window for ARUWPVideo to cast to, performancedegrades severely. This problem was temporarily overcome by castingARUWPVideo's preview to an empty game object containing a mesh. Toovercome this limitation in a more efficient manner, we relocated thecode responsible for what ARUWPVideo does in terms of the preview windowinto the marker tracking routines within the ARToolKit for the HOLOLENSimplementation, which removes the need to create a fake video previewwindow.

The implementation of the ARToolKit library captured webcam footage atthe 1344×756 resolution. The HOLOLENS only supports three resolutionsfor webcam footage capture: 1344×756, 1280×720, and 896×504. Aftertesting, we observed that frame rates were too low with the 1344×756resolution, and the developer of the ARToolKit library advised againstthe 1280×720 resolution. Thus, it was decided to use the lowestresolution of 896×504. This increased frame rates to around 25 fps onaverage. This change in resolution forced us to update the calibrationof the camera stream. We created a custom calibration file usingOPENCV's main calibration method, which entails taking multiple picturesof a checkerboard pattern using the HOLOLENS' camera.

The author of the ARToolKit library, Long Qian of John HopkinsUniversity, provides a program which turns OPENCV calibration files intoARToolKit calibration files, so we then utilized that program to createour calibration files. We then fine-tuned the calibration (magicMatrixand the undistortion function) to help ease out other calibrationissues. This resolution change also implicitly induces a change in thefield of view in which we can track markers. Our previous resolution of1344×756 produced a horizontal field of view of 67 degrees, while ourcurrent resolution of 896×504 produces a decreased horizontal field ofview of 48 degrees.

We also had to make several general improvements to the ARToolKit code.For instance, marker tracking no longer relies on a queue of backloggedtransformations, rather, holograms for marker tracking are alwaysupdated to be the latest transformation of the marker. This helpsholograms to not lag behind the markers when the markers are moved.

In addition, inventors also addressed the orientation-variant markerhologram placement. A single marker is sufficient to place a hologramwith 6 DOF, but multiple marker setups provide redundancy to strengthenthe tracking accuracy and prevent tracking loss, e.g., a 2×2 squaremarker setup with four different ARToolKit marker types usually meansthat at least one or two of the markers is identified in the HOLOLENS'camera image stream.

IMU and Bluetooth

We integrated an MPU6050 inertial measurement unit (IMU) to compensatefor marker tracking loss. MPU6050 devices from InvenSense, a TDK GroupCompany headquartered in San Jose, California, combine a 3-axisgyroscope and a 3-axis accelerometer on the same silicon die, togetherwith an onboard digital motion processor, which processes complex 6-axisalgorithms. In our prototype, this IMU was integrated onto a protoboardwith an ARDUINO Micro and an HC-06 Bluetooth chip.

In order to get IMU data to the HOLOLENS, we utilized a basic BluetoothHOLOLENS

API created in-house at the University of Central Florida, which wasextended, e.g., to clear the Bluetooth message queue in the case that aperformance hitch occurs and the incoming data exceeds the buffer. Wespent some time optimizing the message packet size and reducingprocessing overhead. An ARDUINO Micro was programmed with firmware thatpackages a Bluetooth message, and that firmware is used to send databeing received from code that is processing the IMU data. Translatingthe IMU data to UNITY rotations was straightforward. The IMU sends datain yaw-roll-pitch format, which we replaced with quaternions to avoidGimbal Lock related issues.

IMU Calibration

We calibrated the IMU to the HOLOLENS using a two-step process. First,we performed an intrinsic calibration of the IMU to rule out relativemagnetic interference effects. Second, we designed an extrinsiccalibration process based on an optical marker to align the orientationof the IMU (gravity and magnetic north) and the HOLOLENS'(gravity-aligned) coordinate systems. This process then can also be usedrepeatedly at run time during use to interpret IMU data based off thelast orientation of the IMU before optical image tracking is lost.

We decided to use Quaternions instead of yaw-pitch-roll angles to avoidGimbal Lock issues. We hence changed the ARDUINO firmware that we usedin our prototype to send out data in Quaternion format. However, sincethis increases the data traffic from 3 to 4 values for orientations, wereconsidered this choice if we observed issues related to limitedbandwidth or latency.

IMU and Network Streaming

Several improvements were made in how the data from the IMU is handledin the AR system and in terms of the Bluetooth real-time streaming codefor the 3 DOF orientation tracking data. With regards to the IMU data,we made several changes to how we calibrate the IMU and how we interpretthe IMU data. For instance, we implemented a re-calibration method whichincorporates rotation of the image markers in addition to the rotationof the IMU when visual tracking is lost. This addresses an issue we wereexperiencing where the orientation of a multi-marker hologram would jumpfrom one orientation to another because of differences between theorientation of the marker(s) and IMU.

We made performance improvements to our Bluetooth code to reduce thelatency of IMU data streamed from the instrumented moulage to theHOLOLENS. Updates were thus sent to the HOLOLENS every 30 ms. Pendingformal latency measurements, the results in terms of orientation updateson the HOLOLENS appear more reasonable. We also modified our ARDUINOfirmware to support sending a matrix of values from the IMU and VELOSTATusing a unified protocol. VELOSTAT is a brand name of Desco Industries,Inc. for a packaging material made of a polymeric foil (polyolefins)impregnated with carbon black to make it electrically conductive.

Bluetooth Optimization

Inventors optimized the Bluetooth code. An example being thatpreviously, the Bluetooth code handled a lot of processing within thecode responsible for receiving Bluetooth signals. This reduced howquickly the HOLOLENS software was able to process Bluetooth signals. Tooptimize this, we offloaded all post-message receiving code to outsideof the Bluetooth receiving code. This helped improve Bluetooth signalreceiving speed considerably.

We also have made modifications to both our Bluetooth code and ARToolKitso they work in sync. For instance, when we are relying on imagetracking, the Bluetooth routines lay dormant so they do not take upprocessing power. This specific fix helped reduce graphical stutteringwe were experiencing. Another example being the multi-marker code, whichwas outlined in the previous report, also stops expecting multi-markersand performs no processing relating to such.

Pressure Sensing

The touch and pressure sensitive VELOSTAT surface is designed to havetwo layers of conductive material (one outputting ground, one outputtinga fixed voltage) which sandwich the VELOSTAT. On top of this VELOSTAT“sandwich” a skin-like surface is placed which is made using liquidrubber which represents a moulage which would go over thistouch-sensitive surface.

We integrated code to manage pressure sensing values received from theVELOSTAT material. The VELOSTAT needs to be calibrated against areference value, since voltage values read off the VELOSTAT may bedifferent in different scenarios or environments. To compensate forthis, we calibrate VELOSTAT readings when the HOLOLENS software starts.We accomplish this by reading what the VELOSTAT values are when thesoftware starts up, and then determine pressure values based off itsrelative change. For instance, if the VELOSTAT starts by reading 0.8 andthen decreases to 0.2, then there has been a 75% pressure differential.In the case of a bleeding wound this pressure differential can be usedto determine if a enough pressure has been applied to stop the bleedingbased off of pre-programmed differentials.

To demonstrate the pressure sensing capabilities, we implemented a demousing pressure differentials to reduce the bleeding of an AR wound. Inthis demo, we defined that once the pressure differential reaches 60% tostop the holographic display of blood, and when the pressuredifferential is <=60% to simply adjust the amount of bleeding. Thecloser to 60% the differential is the more the bleeding is reduced.

We also experimented with pressure sensing based on the VELOSTATmaterial.

Physically, this is accomplished by applying isolated regions ofconductive thread throughout the VELOSTAT on one side, and then applyingconductive thread throughout the whole other side of the VELOSTAT (notnecessarily in isolated regions). For the side with isolated regions, weapply a voltage (which is arbitrary, however currently this voltage isapplied by the ARDUINO's PULLUP pins) and on the other side we connectit to a ground. VELOSTAT works by increasing the electrical resistanceof the material when pressure is applied, so when a user presses on theVELOSTAT, a region of pressure can be detected by measuring the voltagedrops in various regions of the VELOSTAT.

Multiple Area Pressure Sensing

In an embodiment of the invention, multiple distinct areas of VELOSTATmay be used to achieve multi-area pressure sensing. One problem thatarises with multi-area pressure sensing is the need for orientationinformation. In order to infer an area of pressure for visualizationpurposes we need to know the orientation of the VELOSTATS. This problemis solved by utilizing our existing marker-and-IMU approach to tracking.

We can deduce the orientation of the VELOSTATS using the orientations ofthe markers and, when that fails, the orientation of the IMU. Note thatthis does require some standardization; for which we assume that when amarker is not rotated at all, we know the default pose of the wound. Weuse this default orientation to then deduce VELOSTAT locations when weare not in default orientation. For instance, in a 2×2 VELOSTAT setup,if we are rotated 90 degrees clockwise according to the markers, thetop-right VELOSTAT becomes the bottom-right, the top-left becomes thetop-right, etc. This kind of deduction is not necessary in setups wherethe VELOSTAT is orientation invariant; however, it is imaginable thatsome VELOSTAT setups may be orientation variant, which calls for a needfor this type of deduction in order to infer the locations of theVELOSTATS, and thus to know if a trainee applied pressure in the rightlocation on or around the wound.

We also had to make changes as to how VELOSTAT values are read in theHOLOLENS software. When the HOLOLENS software was initially created, theBluetooth connection was only used for the IMU. Hence, the IMU codeprimarily managed the Bluetooth data. When we started testing withVELOSTAT, we changed the IMU code to handle the pressure values comingalong with the IMU data. We relay the VELOSTAT data that comes to theIMU code as a result of the VELOSTAT and IMU data being packeted intoone Bluetooth packet. The IMU code takes the IMU data out of theoriginal packet delivered to the code, and then crafts a new packet tobe delivered to the pressure sensing manager containing purely theVELOSTAT information. The pressure sensing manager then deals with thisdata.

A simple system has been implemented where our UNITY application nowstores a simple database of wounds, and each wound has specific markerstied to it. When the HOLOLENS sees a marker, it consults this databaseto attach a certain wound to it. This wound specification is then usedto interpret the VELOSTAT pressure sensing data.

Sensitivity

The VELOSTAT is covered with conductive wire then embedded into thesilicone. We used tape to secure the conductive wire to the VELOSTAT. Wethought that the sensitivity of the VELOSTAT is affected by the shape ofthe wire, so we explored different patterns. It did not turn out to bethe case. Other factors that we would like to explore to find out whatcould affect the sensitivity are the tape used, the size of the patch,or the brand of VELOSTAT. The VELOSTAT has conductive wire. We useddifferent patterns (spiral, parallel) with different spacing. This seemsnot to affect the sensitivity. There is one layer of silicone and onelayer of VELOSTAT. A layer of silicone (pre-coloring) is shaped like agunshot wound and covers the VELOSTAT. We used the transparent wound toshow the VELOSTAT underneath. The same smart wound can have colors.

Physical Moulage

We have been investigating different methods of making moulage, and welooked to develop sample moulage to integrate the VELOSTAT with. Thematerials we were planning to use were SMOOTH-ON'S SMOOTH CAST 300 RESINand SMOOTH-ON'S DRAGON SKIN FX-PRO. The latter of which we used to makevery simple moulage previously.

We investigated different moulage designs. We moved forward with usingSMOOTH-ON products to develop moulage, and we looked more into thatapproach, e.g., shaping non-sulfur clay for the moulage cast, usingSMOOTH-ON'S LIQUID PLASTIC RESIN to cast the shape of the moulage, andusing SMOOTH-ON'S LIQUID RUBBER to finalize the moulage.

In our efforts to develop a more holistic system prototype, we investedsome time on the prototyping of sample moulage. The process has threemain steps: developing a clay mold for a cast, creating the cast, andcreating the moulage from the cast.

The first step requires using non-sulfur clay to model a cast. Thisinvolves shaping what the moulage will look like out of clay, and thenbuilding clay walls around this shape to contain the liquid resin. Oncethe model has been developed, mold release is sprayed onto it. For theresin, we used SMOOTH-CAST 300 LIQUID PLASTIC. We combine the two partsof the liquid resin in a 50/50 mixture, stir, and then pour the liquidplastic onto the clay mold until it fills up to the top of our claywalls. After allowing the liquid resin to set for 10 minutes, we thenremove the hardened resin from the clay. We then proceed to developingthe actual moulage. For the moulage, we used SMOOTH-ON DRAGON SKIN FXPRO LIQUID RUBBER. We combine the two parts of the liquid rubber in a50/50 mixture and stir. If one wished to tint the color of the moulagein a uniform color to give a more skin-like appearance, it should bedone before stirring. After stirring for 90 seconds, we then pour themixture into the cast and let set for 15 to 30 minutes. After the liquidrubber has set, the moulage is removed from the cast. Any imperfectionsin the moulage can be cut or sanded away.

This way of developing moulage is rather straightforward, and while wedid not spend a lot of time attempting to develop more realistic lookingmoulage as such is not in our focus as of now, this process can beenhanced to develop more realistic moulage.

We prototyped a basic moulage. After developing the moulage, we cut andshaped a piece of VELOSTAT into the shape of our moulage and laidconductive thread on both sides of thereof. We then attached our MPU6050IMU to the side of the moulage using an epoxy. We then connect thisIMU-Moulage assembly to a control board consisting of an HC-06 Bluetoothchip and an ARDUINO. We utilized this prototype to test and debug oursystem.

We looked into an improved placement of the IMU module as the pressureapplied to the moulage can move the IMU, causing false orientationchanges. Another area for improvement was the hardware footprint. Acustom board using a small Bluetooth SoC in conjunction with a smallerIMU could reduce the footprint to the point where we could embed theboard in the moulage. The alternative would be to run thin wires from alarger separate control board to the moulage.

Improved Moulage

We developed a moulage where we could integrate the IMU into the moulageas opposed to attaching the IMU to the moulage. This provides improvedorientation accuracy and precision, while also demonstrating how aminiaturized control board with the hardware components could beembedded into the moulage. We also experimented with various initialmethods as to how to embed the VELOSTAT and ARToolKit markers into themoulage.

Moulage Molds and Material Exploration

We explored different mold material and wound shapes. We created rubberand resin molds for a cut wound and a gunshot wound and used these moldsto create moulage using simulated skin from these molds. We used a cutwound as a prototype for the smart-moulage during development.

We created clay and rubber molds for different shapes of wounds. We setup the environment to pour the silicone in. The silicone is degassed ina vacuum chamber then poured into the mold. After the silicone cured, wepainted the top layer with silicone-based paint to make it look like areal wound.

Digital Moulage

We created 3D models for the wounds in AUTODESK MAYA and imported theminto UNITY. In UNITY we can change the shape of the wound and theintensity of blood flow. We connected the change of the wound shape toan XBox controller and connected the blood flow to the smart moulage.

Blood Fluid

We tested and worked on integrating the OBI FLUID UNITY package as arealistic, fluid simulation-based display of blood. However, we foundthat OBI FLUID does not currently work with the Universal WindowsPlatform (UWP), the platform running on the HOLOLENS. We examinedalternatives such as particle-based solutions like the FXVILLE BLOODPACK.

System Prototype Integration

We enhanced our previous prototype in two main ways: We created a biggermoulage so that markers could be placed on the actual moulage itself,and we integrated the IMU as a part of the actual moulage rather thanattaching it on the side of the moulage.

We also refined the reliability of our marker tracking. We adjusted ourprototype to using matrix markers, which when used with the “Mono &Matrix” detection method within ARToolKit provides more reliable markerdetection, and also helps prevent fallacious marker detection. We alsofixed a visualization bug which made holograms appear at the wrongdepth.

We continued working on the hardware to accommodate embedding theelectronics into the mold, worked on improving the appearance of the 3Darterial blood, and experimented with venous blood. We changed theparameters of the blood appearance, flow, angle, and color. We alsoexperimented with increasing the VELOSTAT pressure sensitivity by addingan insulating layer between the VELOSTAT and the conductive wire. Wefurther cleaned up the wires for the IMU, and swapped IMUs to betteraccommodate the moulage demonstration and the IMU evaluation. Theoriginal IMU has a longer wire, the headers are soldered flat and areconnected to the VELOSTAT. The performance evaluation setup has ashorter wire with headers on a 90-degrees angle. We also tested sendinga tactile pulse signal through the moulage by sending an audio file tothe TECHTILE TOOLKIT acoustic-haptic device.

Demonstration and Calibration Application

For the sake of demonstration and for debugging and testing, wedeveloped a more sophisticated demonstration application. We wanted to,for instance, be able to test IMU tracking on-the-fly; something whichpreviously was tedious as it required having to remove the markertracking components from the application. Instead of having to do thisany time we wanted to test a specific component of the prototype system,we developed an application that allows us to select separate scenes fortesting the system. We focused on five different testing scenes: the IMUscene, the ARToolKit/tracking scene, the VELOSTAT scene, the “fusion”scene, and the “demo” scene.

The IMU scene displays a hologram of coordinate axes with the IMU datain the lower left corner and rotates the coordinate axes in accordancewith the IMU data. The ARToolKit/tracking scene projects holograms ofcoordinate axes onto set ARToolKit matrix markers. The VELOSTAT scenedemonstrates a simple VELOSTAT auto-calibration technique and displaysthe pressure values being read from the hardware. The “fusion” sceneshowcases the combined IMU-and-ARToolKit tracking, and the “demo” sceneshowcases all the technologies working in conjunction with an example ofa hologram of blood.

As mentioned previously, in the VELOSTAT scene we implement aproof-of-concept auto-calibration for the VELOSTAT. Thisauto-calibration technique works as follows: the application assumesthat no pressure will be applied to the VELOSTAT for five seconds, andduring that five seconds pressure data is collected from the VELOSTAT,and an average “default pressure” is calculated from the mean of thisdata set. Pressure is then calculated via percent-difference with thatdefault pressure and current pressure.

Other General Improvements

We restructured and organized our UNITY codebase, and we added debuggingfunctionality and automation to the code. For instance, our Bluetoothcode automatically attempts to connect and re-connect to the smartmoulage instead of waiting for user input. In restructuring our code, weimplemented different debug tools. For instance, there is a universal“debug” flag that, when set and triggered, enables console logging to adegree specified by the developer ranging from error logging to verbatimlogs of function calls, Bluetooth messages being sent and received, andwhen ARToolKit finds or loses a marker.

MIXED REALITY TOOLKIT Integration

We integrated MICROSOFT'S MIXED REALITY TOOLKIT, mainly so that ourHOLOLENS software can relay information to a user outside of MICROSOFTVISUAL STUDIO'S debug console. The dialog boxes in the MIXED REALITYTOOLKIT allow us, for instance, to inform a user whenever the Bluetoothconnection fails, or if ARToolKit fails to start due to lack of camerapermissions. If fatal application errors occur, the dialogues in theMIXED REALITY TOOLKIT allow us to report to a user what had occurred,which makes debugging easier for certain bugs that only occur in rarecircumstances. These dialog boxes only appear in developer builds of thesoftware and would not appear if the application is built for releasedeployment. The MIXED REALITY TOOLKIT also presents us with tools thatmay be useful in future versions of the prototype HOLOLENS software. Forinstance, spatial mapping can be useful for helping develop a realisticblood fluid representation, as blood particles can interact with theenvironment around it (e.g., accumulate on the floor underneath thepatient).

Hardware Cleanup

We worked on cleaning up the hardware for our instrumented moulage sothat it can be robustly demonstrated. The first step of this process wasto integrate all our core components on a protoboard instead of abreadboard, and solder all the components together. We also routed allconnections to the IMU and VELOSTAT to a single header found at thebottom of the board, so that we may look towards using a single cable toconnect the instrumented moulage and the board.

We did some research as to what we could use as a small footprintbattery to power this board. We decided to use a 3.7 lithium polymer(LiPo) battery to accomplish this, in conjunction with a combined 5Vstep-up and battery charging component.

We continued to clean up the hardware for our instrumented moulage. Wecontinued the process of integrating the rest of the components andsolder all the components together. We connected a 3.7 LiPo battery inconjunction with a combined 5V step-up and battery charging component.

Performance Evaluation

Informal Evaluation of Marker Tracking Methods

In June we investigated tracking methods, and within our results wenoticed that tracking methods such as VUFORIA'S IMAGETARGET andVISIONLIB'S POSTERTRACKING did not perform well due to the complexity ofthe markers required. Hence, our first goal was to find an imagetracking library that does not require such complex markers. We foundthree of interest: ARUCO, ARToolKit, and VUFORIA'S VUMARKS.

VUFORIA's VUMARKS at first seemed to be the most promising approach.VUMARKS replaced FrameMarkers in recent versions of VUFORIA and seemedto promise a modern API which implemented a marker tracking system thatwe desired. VUMARKS performed better than ImageTargets in terms of beingdetected by a camera. Yet, further testing with this API yielded thatVUMARKS are rather bad at being detected at a distance, as well as invarying lighting conditions.

A HOLOLENS version of ARToolKit was developed by Long Qian at JohnHopkins University and offers a tracking speed of on average 25 to 30frames per second. Upon testing, ARToolKit proved to be the topcontender for what we wanted out of the image tracking library, as itwas able to track markers at a considerable distance, tracked relativelywell in darker lighting conditions, and offered a lot of APIflexibility. ARToolKit also worked well with smaller markers around 1inch in area.

ARUCO, while similar in architecture to ARToolKit, failed to be a viableoption due to its reduced performance on the HOLOLENS. ARToolKit offersa faster tracking speed on the HOLOLENS, and the HOLOLENS version ofARUCO has not been maintained in over two years (e.g., corroborated byLong Qian at John Hopkins University).

Formal Evaluation of ARToolKit versus VUFORIA

We performed a comparative evaluation of ARToolKit and VUFORIA using theexperimental method and setup that we had developed earlier. We set up atracking environment consisting of a Microsoft HOLOLENS and aprofessional optical tracking system (OPTITRACK Duo), the latter ofwhich provides sub-millimeter precision and accuracy. We tested the mostrecent versions of VUFORIA and ARToolKit as described above.

An experimental setup at UCF was assembled showing the HOLOLENS set upon a Styrofoam head, distance indicators on the table in front of theHOLOLENS, where we tested visual markers (VUFORIA and ARToolKit) atdifferent distances, while we also attached an OPTITRACK 6 DOF target tothe markers to track them with a professional high-precision opticaltracking system. An OPTITRACK 6 DOF infrared retro-reflective targetswas attached to ARToolKit (left) and VUFORIA (right) markers. We chosethe ARToolKit marker as it was the default marker for this trackingsystem. We made sure that the passive infrared parts did not affect theRGB camera of the HOLOLENS that both VUFORIA and ARToolKit use. Bothmarkers had a size of 5 cm×5 cm with a white frame of 1 cm.

We compared the tracked distances provided by ARToolKit and VUFORIA atdistances from 0.33 m to 2.33 m in steps of 0.33 m. We tested the twosystems one after another using the same HOLOLENS, since both cannotaccess the HOLOLENS' camera at the same time, and we wanted to avoiderrors potentially caused by using different HOLOLENS units or differentframes of reference. At each distance, we positioned the marker facingstraight at the HOLOLENS and we held it in our hand, while moving itslowly (up/down and left/right) and rotating it slowly (yaw, pitch,roll) in a range of +/−5 cm and +/−15 degrees. We kept the center of themarker always at the same distance from the HOLOLENS for each of theseven distances as indicated on the table. The process was checked forvalidity considering ground truth provided by the OPTITRACK professionaltracking system.

FIGS. 40-41 show the results we received from the two marker trackingsystems. The x-axis showed the actual distance of the marker and they-axis shows the tracked distance as reported by ARToolKit (green) andVUFORIA (blue). The tracking performance of ARToolKit is much betterthan VUFORIA over longer distances, while it is hard to see a cleardifference over shorter distances of up to a meter. In fact, we couldnot collect any data for VUFORIA beyond 1.33 m since no markers weredetected at these distances. ARToolKit, in contrast, was reasonablystable over all tested distances. In particular, we believe that the newversion of ARToolKit made it more robust than the previous versions,probably due to them switching to computations based on grayscale imagesinstead of VUFORIA's approach to use nicer-looking colored images thatare harder to track over longer distances.

The results of the comparison between VUFORIA (blue) and ARToolKit(green) are shown in FIGS. 14 and 15. The error bars show the standarddeviation. In FIG. 14, the x-axis shows the actual distances; the y-axisshows the tracked distances. FIG. 15 provides a detailed plot of therelative differences (offset errors) between the tracked distances andtheir actual distances. The results on the y-axis indicate that VUFORIAprovides reasonable results for closer distances, while it shows largeerrors at 1.33 m, and does not provide results beyond that distance.ARToolKit shows reasonable results for the entire tracked range. Meanerror offsets are below 0.1 m, while the offsets and standard deviationsincrease over distance.

FIG. 16 shows the tracking performance in terms of frames per second inwhich we received tracking data from VUFORIA and ARToolKit. As statedbefore, we did not receive any tracking data for 1.66 m distance orbeyond for VUFORIA. We did not expect to see ARToolKit providing suchreasonable data for the longer distances. FIG. 16 demonstrates overalltracking frames-per-second for VUFORIA (blue) and ARToolKit (green).Both did not reach 30 fps. VUFORIA did not provide any tracking data for1.66 m and beyond. ARToolKit provided more data over longer distances.

Overall, our test data suggests that the new version of ARToolKitprovides reasonably good tracking data for a visual marker trackingsystem. It does not work with colored markers and image markers as doesVUFORIA, but, considering the quite reasonable results, we believe thatit is worth using ARToolKit since VUFORIA is only reasonably stable forup to one meter. If, for instance, the casualty (mannequin) is lying ona table, the VUFORIA system should work well, while it would lead tolarge tracking errors if the casualty is lying on the floor and the useris standing or kneeling at its side. However, the tradeoff is that themarkers will not look as good as with VUFORIA.

Formal Evaluation of ARToolKit versus IMU

We designed a study to evaluate the IMU in comparison to marker-basedtracking (i.e. ARToolKit) and ground truth as measured by an OPTITRACKV120:TRIO, a professional IR-based tracking system.

Testing and Initial Study Design

Our test tracked device incorporated all three tracking sources in asingle tightly coupled single object so that we would be able to collectsynchronized, comparable data from each tracking method simultaneously.To this end, we used a standard configuration OPTITRACK rigid bodymarker comprising four retroreflective spheres, an ARToolKit marker inthe center of the plane of the OPTITRACK markers, and finally we rigidlyaffixed the IMU to the center of the back, directly behind the ARToolKitmarker. We also added some simple physical shafts for the ability tomanually rotate the objects in each of the 3 basis axes (X, Y, Z)independently.

We tested the orientation data streaming for the setup with sevenecologically useful distances. Using OPTITRACK's Motive software, therigid body tracking data is streamed via the network to UNITY andrunning on the HOLOLENS to perform the ARToolKit tracking as well asreceive the IMU data via Bluetooth. This includes an additional step toconvert the OPTITRACK rigid body 6-DoF pose from Motive's coordinatesystem to UNITY's. We are still in the process of implementing anecessary calibration step for adjusting the IMU rotation data to thesame common reference frame as the ARToolKit and OPTITRACK data.

Finalized Evaluation Design

The idea behind this study was to understand the uses of an IMU as ameans to support optical marker tracking (e.g., ARToolKit or VUFORIA),and to compare the performance in different tracking situations. With a9-axis IMU based on 3-axis gyroscopes, accelerometers, andmagnetometers, we expect to see more stable and accurate orientationinformation compared to optical marker tracking, in particular due tothe fact that the IMU is unaffected by occlusion and lighting issues. Wedescribed the initial design for this study in last month's report. Weperformed the IMU evaluation in March. In this section, we present thematerials, methods, results, and discussion.

Materials

We performed the study at our SREAL laboratory at UCF using anexperimental setup consisting of the following components:

-   -   OPTITRACK V120: Trio, professional infrared (IR) based tracking        system, consisting of three calibrated IR cameras, mounted on a        tripod—this system provided “ground truth” pose measurements,    -   Microsoft HOLOLENS head-mounted display, mounted on a Styrofoam        head,    -   ARDUINO Micro, mounted on a rigid wooden plate, to which we        attached an Adafruit BNO-055 IMU, a standard 5 cm×5 cm ARToolKit        marker, and a predefined OPTITRACK rigid body with four        retro-reflective markers.

On the software side, we used our latest running UNITY version2017.4.8f1 environment with the most recent ARToolKit for HOLOLENSversion to track the marker with 6-DOF tracking data. We streamed theIMU's 3-DOF tracking data from the ARDUINO to the UNITY applicationrunning on the HOLOLENS using our previously developed Bluetooth code.We further streamed the OPTITRACK 6-DOF tracking data from a dedicatedserver workstation to UNITY. We timestamped the tracking data byOPTITRACK, ARToolKit, and IMU, and logged everything in UNITYcoordinates.

An ARToolKit marker, OPTITRACK 6-DoF target (retro-reflective markers),and IMU (attached to the ARDUINO) were mounted together as a rigid bodyfor the performance evaluation.

Methods

We used a 2×3×7 design for this performance evaluation. Our factors aredescribed in the following:

-   -   Tracking method (2 levels): ARToolKit and IMU    -   Rotation direction (2 levels): pitch, roll, and yaw    -   Distance to HOLOLENS (7 levels): 0.33, 0.66, 1, 1.33, 1.66, 2,        2.33 meters

The conditions are described in detail below.

Tracking Method.

We used either the ARToolKit optical marker tracking with the HOLOLENS'color camera to track the 5 cm×5 cm standard marker, or the IMU mountedrigidly on the same marker. As ground truth, we calibrated each of themto a professional OPTITRACK tracking system, which is hence not listedas a separate tracking method but rather is the standard to which wecompared the two other methods to assess their accuracy.

Rotation Direction.

Starting from an orientation in which the marker was facing directly atthe HOLOLENS, see FIG. 9, i.e., in full view of the HOLOLENS' camera, werotated the marker around each of the three axes (X, Y, Z as shown inthe figure) by two full 360-degree rotations. By performing theserotations, we hence tested the entire range from 0 to 360 degrees forall three orientation directions pitch (X), yaw (Y), and roll (Z). Forpitch and yaw rotations, it is technically impossible for thecamera-based ARToolKit marker tracking to provide data beyond a90-degree rotation (clockwise or counterclockwise) from the startorientation facing towards the HOLOLENS. For roll rotations, however,this limitation is not present as the marker is always facing towardsthe HOLOLENS. The IMU is not affected by the self-occlusion of themarker, hence we expect to get full 360-degree data for each of thethree axes for the IMU.

FIG. 17 is an illustration of the rotation directions showing theARToolKit marker. We started from an orientation in which the marker wasfacing straight at the HOLOLENS in full view. We then rotated the markeraround one of the three axes in space. We rotated the marker by two full360-degree rotations around each of the three axes (i.e., X, Y, Z).

Distance to HOLOLENS. Using the same experimental setup as in theprevious comparison of ARToolKit and VUFORIA we tested a maximum rangeof up to 2.33 meters from the HOLOLENS. We started at a distance ofone-third of a meter and tested the range in steps of 0.33 meters (i.e.,0.33, 0.66, 1, 1.33, 1.66, 2, 2.33 meters). We expect the trackingaccuracy of ARToolKit to decrease for increasing distances to theHOLOLENS' camera, while the IMU data should be unaffected by thedistance to the HOLOLENS; that is, as long as it remains within tenmeters Bluetooth range to the ARDUINO.

Procedure and Analysis

For each of seven distances to the HOLOLENS we held the rigid body (withthe ARToolKit marker, IMU, and OPTITRACK markers) at the distanceindicator. We started the logging via a button press on an Xboxcontroller paired with the HOLOLENS. For each of the three rotationdirections, indicated in FIG. 17, we then performed two full 360-degreerotations by slowly rotating the rigid body by hand around thecorresponding rotation axis. Each of those was performed over aninterval of approximately 30 seconds. We logged the tracking data fromall three systems (ARToolKit, IMU, and OPTITRACK) for these rotations atthe same time and we saved it to a log file on the HOLOLENS.

We analyzed the performance in Matlab based on the logged informationfor the two tracking methods, two rotation directions, and sevendistances to the HOLOLENS. As expected, ARToolKit provided far fewertracking data points than the IMU, such that we opted for the followinganalysis. We calibrated the orientation data for ARToolKit and the IMUseparately to the OPTITRACK system using their initial coordinateframes. For each new tracking data point provided by ARToolKit or theIMU, we looked up the closest timestamped data point from theprofessional OPTITRACK tracking system. We considered the OPTITRACKperformance as the ground truth in this evaluation. If there was no datapoint for OPTITRACK within the limits of its frame rate, we excluded thedata point from the analysis.

The results shown in the following section thus provide indications onthe accuracy of the tracking data points provided by ARToolKit and theIMU, while their frequency is dependent more on the sampling frame rateof the HOLOLENS' camera, Bluetooth, etc.

Results and Discussion

FIGS. 49-54 shows the results for the different tested conditions. Theplots are designed as hybrid scatter plots, showing the individualresults in different colors, as well as the means and standarddeviations. Additionally, Table 1 lists the mean results in thedifferent conditions for ARToolKit. Table 2 lists the correspondingresults for the IMU.

As expected, the IMU results show the expected distribution and patternof angular errors that are independent of the distance to the HOLOLENS.While the IMU's Bluetooth connection may result in packet loss overlonger distances (e.g., ten meters), we noticed no such effects over thetested distances. Considering the axes, shown in FIG. 41, with respectto the strengths and limitations of the 9-axis IMU, with 3-axisaccelerometers, gyroscopes, and magnetometers, we observed that the yawerrors (M=3.1 deg) were lowest, followed by pitch (M=6.0 deg), and rollerrors (M=8.4 deg). Overall, our results show that this particular IMUperformed reasonably well.

The ARToolKit results show the expected distribution of angular errorsthat increase for increasing distances to the HOLOLENS' camera, inparticular when the marker surface is angled towards the HOLOLENS. Forthe tested roll rotations, we observed a more consistent pattern oferrors over the tested range of distances, which is mainly related tothe point that the marker was always faced straight at the HOLOLENS'camera, such that the errors were mainly affected by its resolution andthe image space of the marker. For roll rotations, the trackingdifficulty was hence comparatively low. In contrast, for the pitch andyaw rotations, we see that the errors increase largely for largerdistances to the HOLOLENS. The main causes for this effect additionallyto the reduced image space are the angled marker and differences inrelative lighting (e.g., light in the real world usually comes fromabove not below). The distances had a strong effect on angular errors,which increased from the closest tested distance of 0.33 meters (pitch:M=1.26 deg; yaw: M=1.30 deg) to 2.33 meters (pitch: M=13.49 deg; yaw:M=17.07 deg). In addition to the increase in mean errors for largerdistances, we further see that the variance and magnitude of outliersincrease.

FIGS. 55-60 further show the same results here in the form of signederrors (positive or negative angular differences relative to theOPTITRACK tracking data used as ground truth orientations). While weobserved that angular errors for some of the distances might suggest apattern of more positive or negative signed angular differences(mathematically positive/negative rotations), as expected, we did notfind a consistent pattern over all distances, i.e., the apparentrelative differences among the tested distances might be explained bysubtle changes in lighting conditions or differences in feature pointsidentified through ARToolKit's internal grayscale image processing andtracking algorithms.

In summary, the ARToolKit results are reasonably good when the marker isoriented straight at the HOLOLENS' camera and/or placed very close tothe HOLOLENS. In comparison, the IMU shows far better results for longerdistances, both in terms of fewer extreme outliers as in terms of theoverall mean errors. Moreover, due to the 9-axis sensors in the IMU, wedid not see error propagation or drift over the course of theevaluation, as would be typical for pure 3-axis accelerometers orgyroscopes.

TABLE 1 Mean angular errors when ARToolKit provided tracking data forthe marker. ARToolKit provided data only for a rotation of the marker ofup to +/−60 degrees facing away from the HOLOLENS. These values onlyconsider the errors compared to the ground truth when a marker wasactually tracked. Distance: Cond: 0.33 0.66 1 1.33 1.66 2 2.33 MeanPitch Error: 1.26 2.73 4.60 6.91 9.98 12.16 13.49 Mean Roll Error: 6.938.76 6.03 6.42 6.45 6.28 7.30 Mean Yaw Error: 1.30 2.89 5.09 9.75 14.4411.31 17.07

TABLE 2 Mean angular errors when the IMU provided tracking data for themarker. The IMU provided data for the entire 0-360 degree range for eachof the three axes, independently of the distance to the HOLOLENS.Distance: Cond: 0.33 0.66 1 1.33 1.66 2 2.33 Mean Pitch Error: 5.73 5.915.97 6.14 6.17 6.31 5.86 Mean Roll Error: 10.45 7.02 7.98 8.40 8.49 8.078.44 Mean Yaw Error: 4.07 2.36 4.75 3.85 2.44 2.13 2.29

Formal Evaluation of ARToolKit Marker Sizes

The idea behind this evaluation was to test different marker sizes usingthe optical marker tracking system that had shown better results in theprevious evaluation, i.e., using ARToolKit instead of VUFORIA. Since thetracking quality of the 5 cm×5 cm marker used in the previous ARToolKitevaluation already provided comparatively good results over a distanceof 2.33 meters, we were looking specifically at trying to understandwhat happens when we reduce the size of the marker. In the following, wepresent the materials, methods, results, and discussion.

Materials

We performed the study using the experimental setup for the IMUevaluation described above. We used the same hardware and softwaresetup. However, we varied the rigid body and size of the ARToolKitmarker. In particular, we mounted the marker on a protractor with afixed base to stand on the surface.

Inventors aimed for static orientations and not dynamic rotations, suchthat we opted to use a static setup with a protractor underneath theARToolKit marker. We rotated it from −60 degrees to +60 degrees in30-degree increments.

Material

We used a 4×5×7 design for this marker size evaluation. Our factors aredescribed in the following:

-   -   Marker size (4 levels): 5 cm, 4 cm, 3 cm, and 2 cm (square        markers; marker width matched its height)    -   Marker orientation (5 levels): −60 deg, −30 deg, 0 deg, 30 deg,        60 deg (angles indicating yaw orientations; 0 degrees indicates        that the marker is facing straight at the HOLOLENS)    -   Distance to HOLOLENS (7 levels): 0.33, 0.66, 1, 1.33, 1.66, 2,        2.33 meters

Details on the conditions are provided below. We used the same distancesas for the IMU performance evaluation described above.

Marker Size.

We used markers of size 5 cm×5 cm previously as a tradeoff betweentracking accuracy and size, which showed reasonably accurate results inthe previous studies. In pilot tests, we further observed that markerswith a width and height of just 1 cm did not provide stable trackingperformance over the described distances. Hence, we decided to testmarker sizes between 5 cm and 2 cm in steps of 1 cm.

Marker Orientation.

In the orientation evaluation comparing the ARToolKit and IMU solutions,we found that the system mainly provided data points for an angle of upto approximately 60 degrees from facing straight at the HOLOLENS'camera. Hence, in order to understand how the marker size affectedtracking performance over this range, we chose to test them in discretesteps. We opted to test them in steps of 30 degrees from −60 to +60degrees. Due to the similarities observed between the performance forthe rotation axes in the comparative IMU-ARToolKit performanceevaluation above, we chose to focus on yaw orientations over this range.

Procedure and Analysis

For each distance, we placed the rigid body at the correspondingdistance from the HOLOLENS. We then oriented it to match the intendedorientation (and precise distance) from the HOLOLENS using a livepreview of calibrated tracking data provided by the OPTITRACKprofessional IR based tracking system. We then started the logging ofthe ARToolKit tracking data on the HOLOLENS using a paired Xboxcontroller. We recorded tracking data for a total of three seconds percondition. We saved the log files on the HOLOLENS. We analyzed theresults in Matlab by computing the tracked distances (i.e., we were notfocusing on orientations as these were covered mainly in the previousevaluation and suggested large performance benefits for the IMU overARToolKit) and the frame rate provided by ARToolKit running on theHOLOLENS in the different conditions.

Results and Discussion

FIGS. 30-33 show the results for the different conditions. The x-axesshow the actual distance of the marker from the HOLOLENS, and the y-axesshow the tracked distance provided by ARToolKit. The plots show themeans and the error bars show the standard deviations. We did not seeany difference in the results for negative and positive angles anddecided to pool the results for these (symmetrical) angles. Hence, thecolored lines show the results for 0 degrees, 30 degrees, and 60 degreesyaw angles to the HOLOLENS, with the marker facing straight at theHOLOLENS in the 0-degree condition. Not all the orientations and markersizes result in tracking information provided by ARToolKit. If no datais provided for one of them, it indicates that we did not receive anyoutput from ARToolKit.

Table 3 shows the ARToolKit frame rates for the different marker sizes,orientations, and distances. As can be seen, we did not receive trackingdata points for some of the conditions. Overall, the frame rate wasreduced and the probability for missing data points was increased forsmaller marker sizes, and increased angles with the marker pointing awayfrom the HOLOLENS.

Distance: Cond: 0.33 m 0.66 m 1 m 1.33 m 1.66 m 2 m 2.33 m 5 cm  0degrees 24.2 28.8 28.2 28.6 28.6 20.0 25.1 30 degrees 22.7 30.0 29.529.6 28.7 14.8 15.5 60 degrees 25.9 30.0 29.0 0 28.6 0 0 4 cm  0 degrees28.2 27.1 29.5 29.2 28.8 17.2 28.8 30 degrees 29.1 28.4 27.1 28.7 30.015.4 19.5 60 degrees 26.8 29.7 28.5 27.7 26.5 0 0  0 degrees 24.9 12.625.3 30.0 15.3 17.0 19.2 3 cm 30 degrees 26.2 30.0 30.0 26.9 19.0 20.114.7 60 degrees 27.7 28.3 28.0 28.3 29.3 28.4 0 2 cm  0 degrees 29.426.7 27.0 28.2 12.8 0 0 30 degrees 24.2 29.9 18.4 0 10.6 0 0 60 degrees21.4 0 12.1 0 0 0 0

The results show that, matching the results from our previouscomparative evaluation between ARToolKit and VUFORIA, the performancefor the 5 cm×5 cm standard marker was comparatively good. For smallermarkers, the errors in distance measures and the number of missing datapoints increased. For the 2 cm×2 cm marker, we did not receive any datapoints for the distances of 2 or 2.33 meters, even when the marker wasfacing straight at the HOLOLENS. Moreover, smaller markers showedincreased problems tracking the marker when it was angled away from theHOLOLENS.

In general, we found that markers of 5 cm×5 cm were necessary to providereasonable tracking data over the entire 2.33 meter range, while markersof size 4 cm×4 cm and in part also markers of size 3 cm×3 cm could betracked reliably over shorter distances of up to one meter. Markers ofsize 2 cm×3 cm only provided tracking data when very close to theHOLOLENS and when they were not angled away from the HOLOLENS' camera.

FIGS. 30-33 show results of the marker size evaluation. The plots showthe results for 5 cm, 4 cm, 3 cm, and 2 cm markers, respectively. Thex-axes show the actual distances of the markers, while the y-axes showthe tracked distances. Optimal results would be aligned along thediagonal from the lower left to the upper right. We received no trackingdata for some of the distances and orientations, which is shown indetail in Table 3.

Video Recordings

Year 1 Demonstration Video Recordings

In preparation for the video recordings, we cleaned up the hardware,created a larger moulage to fit on an arm or limb, embedded thetechnology into the mold, created 3D wounds and 3D blood. The size ofthe moulage can accommodate for using the full hand when applyingpressure (instead of one finger). The smart moulage sends data to theHOLOLENS in which the software dynamically changes the visuals of theblood and of the 3D wound.

We prepared and shot a video at the University of Central Floridashowing Dr. Frank Guido-Sanz using the prototype. We mounted onecamcorder on the HOLOLENS, set up studio lights and camcorders ontripods, and one mobile camcorder in our professional HuSIS space in theSREAL laboratory to capture the action from different angles. We alsoset up a black background on the floor and on the tripods to avoidbackground distractors. We recorded videos with Dr. Frank Guido-Sanzwhere the moulage was positioned on a mannequin arm and a real human.The videos featured changing between the mannequin and the real humanwhere the arm gets moved and the AR imagery follows. We also varied theshape of the wound (gunshot versus shrapnel) and varied whether Dr.Guido-Sanz was wearing a HOLOLENS. After the recording session, wesynchronized the videos and edited a combined video.

We pilot tested different locations around our lab at different times ofday and set up an outdoors video shoot with a live actor. We were ableto show an outdoor view through the HOLOLENS (top) using random 3Dobjects to test the colors and brightness on the HOLOLENS, and (bottom)showing the augmented moulage on a live actor.

Glossary of Claim Terms

Augmented Reality (AR) means a form of virtual reality that augments auser's view of the real world with virtual content. Various mechanismsexist for carrying out the augmentation. One popular mechanism is tolayer virtual information over a live camera feed of the real world fedinto a headset known as a video see-through head-mounted display (HMD),possibly through a portable computing device. Alternatives include theuse of optical see-through HMDs where the virtual content is layeredover a direct view of the real as seen through an optical combinersystem (combining real and virtual views); handheld augmented reality,for example via a mobile “smart” phone with a camera and a display; andspatial augmented reality (SAR) where the virtual imagery is displayeddirectly on (or as part of) real world objects, as opposed to in theuser's visual field. One popular means for SAR is to use a digitalprojector to project imagery directly on an object, hence changing theappearance of the object. One specific method is that of Shader Lampsrendering (Raskar, Welch, and Fuchs), which is sometimes referred to as“projection mapping.” These are just some examples of ways to achieveAR.

Communicatively Coupled means a data connection between one or morecomputing, sensor, storage, and/or networking devices whereininformation is exchanged.

Computer Simulation Engine means the combination of computerprocessor(s) and software instructions to coordinate a medicalsimulation integrating both physical and computer-generated reactions tothe actions of a trainee as detected by sensors and/or extrinsiccontrols (e.g., instructor control of simulation).

Corporeal means a tangible, physical object.

Dimensional means having enough depth and substance to be believable.

Haptic means the generation of a sense of touch by applying motions,vibrations, changes in temperature, changes in texture, changes inmoisture or forces to a user.

Headset or Head-Mounted Display (HMD) is a head-worn device thatprovides virtual reality to the wearer. They often include astereoscopic pair of display screens (frequently providing separateimages for each eye), stereo sound capability, and head motion positionand/or orientation tracking sensors, e.g., employing gyroscopes,accelerometers, or structured light systems.

Indicia (or Indicium) means signs, indications, or distinguishing marks.For the purposes of claim construction, an indicium (singular) does notpreclude additional indicium (e.g., indicia or multiple orientationmarks).

Manikin, a variation of “mannequin,” is a term referring to the physical(typically robotic) form of the simulation of a human, typically jointedbut may include only a portion of an entire body such as limbs, torso orthe like. For the purposes of this invention, manikin may includenon-human animal models useful for veterinary training.

Moulage means the process of applying mock injuries for the purpose oftraining medical personnel. Moulage may range from applying simplerubber or latex “wounds” over an actor portraying a patient to complex,simulated manikins having elements of realism such as vomit, blood, openfractures, and the like.

Optical means operating in or employing the visible part of theelectromagnetic spectrum.

Sensor means a device or peripheral thereof that detects or measures aphysical property and records, indicates, or otherwise responds to it.

Tactile means perceptible to touch.

Virtual Reality means a computer-generated scenario that simulates arealistic experience. Augmented reality systems are considered a form ofVR that layers virtual content into the user's visual field using videoor optical HMDs, smart phones, or directly in the scene as describedabove, optionally through a portable computing device.

Vital Signs means a group of important signs that convey the status ofthe body's vital functions. Examples of five primary vital signs includebody temperature, blood pressure, heart rate, respiratory rate, andpain.

Wound means an injury to living tissue caused by a cut, blow, or otherimpact, typically one in which the skin is cut or broken but may alsoinclude those under intact skin. A wound may further encompass damage toliving tissue from chemical, biological or environmental exposures.

The advantages set forth above, and those made apparent from theforegoing description, are efficiently attained. Since certain changesmay be made in the above construction without departing from the scopeof the invention, it is intended that all matters contained in theforegoing description or shown in the accompanying drawings shall beinterpreted as illustrative and not in a limiting sense.

What is claimed is:
 1. A tactile, augmented reality wound simulationsystem comprising: a corporeal object comprising a simulated wound,wherein the corporeal object is deformable by physical manipulationsimulating physical, tactile and dimensional features of the wound,wherein the wound is on a body surface of the corporeal object, thecorporeal object initially in a first physical state; a visual graphicenhancement of the wound on the body surface of the corporeal object,the visual graphic enhancement renderable via an augmented realitymechanism selected from the group consisting of video see-throughheadsets, optical see-through headsets, handheld augmented realitydevices, and spatial augmented reality devices, the augmented realitymechanism producing an augmented reality rendering; whereby the visualgraphic enhancement overlays the wound visually on the corporeal objectvia the augmented reality mechanism; one or more reference indiciadetectable by an optical sensor, the indicia fixed to the surface of thecorporeal object; and a computer processor communicatively coupled tothe optical sensor and the augmented reality mechanism, the computerprocessor executing a software process using the indicia on thecorporeal object to dynamically locate, size and orient the visualgraphic enhancement rendering of the wound on the corporeal objectwhereby changes to positions or orientations of the one or morereference indicia caused by physical manipulation of the corporealobject to a second state invoke the software process to reconstructspatial and dimensional parameters of the visual graphic enhancement sothat there is a synchronization of the augmented reality rendering tothe second state of the corporeal object.
 2. The system of claim 1further comprising a plurality of predefined physical states of thecorporeal object detectable by position of the indicia by the opticalsensor wherein a first visual graphic enhancement within a plurality ofvisual graphic enhancements is associated with the first state of thecorporeal object and a second visual graphic enhancement is associatedwith the second state of the corporeal object whereby the transitionfrom first to second visual graphic enhancements aligns withphysiological characteristics of the wound responsive to deformation ofthe corporeal object.
 3. The system of claim 1 wherein the physicalmanipulation is selected from the group consisting of rotation,elevation, articulation, compression, expansion, stretching, andpulling, the physical manipulation changing the location or orientationof the indicia whereby the software process, using the indicia, adaptsthe visual graphic enhancement to a new topology of the corporeal objectresponsive to the physical manipulation.
 4. The system of claim 1further comprising a plurality of predefined physical states of thecorporeal object detectable by one or more sensors on or within thecorporeal object, the sensors communicatively coupled to the computerprocessor wherein a first visual graphic enhancement within a pluralityof visual graphic enhancements is associated with the first state of thecorporeal object and a second visual graphic enhancement is associatedwith the second state of the physical corporeal object whereby thetransition from first to second visual graphic enhancements isconsistent with physiological characteristics of the wound responsive tosensor data.
 5. The system of claim 4 wherein the one or more sensors onor within the corporeal object detects a value selected from the groupconsisting of user proximity, mechanical pressure, temperature, sound,articulation and light.
 6. The system of claim 1 further comprising acomputer processor-controlled physical response to changes of theindicia positions or indicia orientations caused by the physicalmanipulation of the corporeal object, the physical response invoked byan electronically actuated device causing the physical response selectedfrom the group consisting of movement, sound, and fluid flow.
 7. Thesystem of claim 5 further comprising a computer processor-controlledphysical response to changes in the sensor value, the physical responseinvoked by an electronically actuated device causing the physicalresponse selected from the group consisting of movement, sound, andfluid flow.
 8. The system of claim 1 further comprising a 3D model ofthe corporeal object accessible by the computer processor wherein thesoftware process reconstructs the spatial and dimensional parameters ofboth the visual graphic enhancement and the 3D model of the corporealobject for the augmented reality mechanism wherein an optical image ofthe corporeal object is spatially and dimensionally aligned to the 3Dmodel wherein the 3D model is rendered opaque or semi-transparent viathe augmented reality mechanism.
 9. The system of claim 1 furthercomprising a glove indicium on a trainee-worn glove that engage thecorporeal object and wound, the computer processor further accessing a3D glove model wherein the glove indicium conveys the orientation,location and articulation of a trainee hand wearing the glove todynamically align the 3D glove model with the trainee hand, wherein thecomputer processor chroma-keys the trainee-worn glove so that an opticalimage opacity for the 3D glove model is modified.
 10. The system ofclaim 1 further comprising glove indicia on the glove that engages thecorporeal object and wound, the computer processor further generating avisual graphic enhancement over the glove simulating fluid.
 11. Thesystem of claim 1 further comprising a vital sign readout wherein thefirst and second states of the wounds simulation system comprise achange in the vital sign readout.
 12. The system of claim 1 furthercomprising a computer processor-controlled response by the physicalcorporeal object to a sensor-detected action.
 13. The system of claim 5wherein a response by the physical corporeal object is selected from thegroup consisting of movement, sound, and fluid flow.